New features

Let's start by discussing the new NSD features in Notes and Domino 8.5.2.

In the event of an notes2 abnormal termination, the callstack that calls panic() will no longer be marked fatal (NBRR83LM3U)

Under the Notes Standard Client, the nlnotes process periodically checks to ensure that the notes2 process is still alive. If the process exits unexpectedly, then the nlnotes client will panic, resulting in an NSD with the following callstack:

############################################################
### thread 7/26: [ NLNOTES: 0a5c: 091c] FATAL THREAD (Panic)
### FP=0x0525dbb4, PC=0x7c90e514, SP=0x0525db50
### stkbase=0x05260000, total stksize=262144, used stksize=9392
### EAX=0x00000000, EBX=0x0525e784, ECX=0x7ffd5000, EDX=0x00270608
### ESI=0x000006f0, EDI=0x00000000, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297
############################################################
[ 1] 0x7c90e514 ntdll.KiFastSystemCallRet+0 (6f0,493e0,0,525e364)
[ 2] 0x7c802532 kernel32.WaitForSingleObject+18 (6f0,493e0,525eaf8,525e7a9)
@[ 3] 0x601b4ec5 nNOTES.FRSendCommandToService+789 (525e384,525e784,525e7a9,0)
@[ 4] 0x601b5aff nNOTES.OSRunExternalScript@8+1055 (12c,1)
@[ 5] 0x601b607f nNOTES.FRTerminateWindowsResources+975 (1,1010,1,0)
@[ 6] 0x601b64a8 nNOTES.OSFaultCleanupExt@24+984 (c76a68,1010,0,0,0,525ee20)
@[ 7] 0x601b652a nNOTES.OSFaultCleanup@12+26 (0,1010,0)
@[ 8] 0x601c1a24 nNOTES.OSNTUnhandledExceptionFilter@4+276 (525fe58)
@[ 9] 0x6018477d nNOTES.Panic@4+589 (525fe70)
@[10] 0x60184834 nNOTES.OSWCTPanicShutdown+84 (a74,1232ee8,12357ee,3)
@[11] 0x63680da3 nnotesws.WCTShutdownThread+211 (a74,138390,7c910041,0)
@[12] 0x6010dabf nNOTES.ThreadWrapper@4+175 (0)
[13] 0x7c80b699 kernel32.GetModuleFileNameA+442 (0,0,0,0)

The above callstack is marked fatal because it invoked the panic routine. However, it is not representative of the underlying problem, which was the unexpected termination of the notes2 process.

To avoid confusion this call stack is no longer marked fatal in Notes 8.5.2.

Adds support for locating the Notes.ini automatically (DCOY6QLLDC)

On the Microsoft® Windows® platform, NSD is now able to determine the location of the Notes.ini for the current user from the registry. NSD searches for a Notes.ini or for an ini.nbf, which contains a path to a Notes.ini file, following these rules:

1. If the -ini switch was specified as a command line argument to NSD, then use this value as the Notes.ini.
2. If there's a NOTES_DATA_DIRECTORY environment variable, look there first for the Notes.ini or ini.nbf file. This environment variable is not intended to be set by the user. In previous versions of Notes, this environment variable would be set by Notes processes on startup, so that NSD could easily locate the data directory at crash time.
3. If the Notes.ini or an ini.nbf file is present in the current directory.
4. If NSD is running from a Notes client installation, look In the registry.

If this is a multiuser Notes Client installation, then NSD will look in HKEY_CURRENT_USER\Software\Lotus\Notes\8.0 for a key called NotesIniPath, which contains the path to the Notes.ini for the current user's Notes installation.

If it's not a multiuser installation, or if the Notes.ini for the multiuser client install could not be found, then look in the directory specified in HKEY_LOCAL_MACHINE\Software\Lotus\Notes\8.0\DataPath (the path to the data directory for a single user install) or Path (the program directory).

NOTE: NSD determines whether the current installation is a Notes Client or Domino server installation by looking in the directory that contains nsd.exe for the presence of nlnotes.exe and/or notes2.exe, and the absence of nserver.exe. If nserver.exe is found, then NSD will not look in the registry for the Notes.ini, but rules 1 through 3 above will still apply.

On Linux® and Macintosh there is no registry to query; however, NSD attempts to locate the Notes.ini using the following rules:

1. If the -ini switch was specified as a command line argument to NSD, then use this value as the Notes.ini.
2. If the Notes.ini is present in the current directory.
3. If running in a Notes Client installation, then look in the typical user-specific locations for the Notes.ini.

Under Linux, NSD looks for the Notes.ini under $HOME/lotus/notes/data.

Under OS X, NSD looks for the Notes.ini under /Users//Library/Preferences/Notes Preferences.

NOTE: Under Linux, NSD checks whether we are running out of a client installation by checking for the presence of lnotes and/or notes2, as well as the absence of server. Under OS X, NSD always assumes a client installation.

Display keyview version information inside NSD (SMAI78VQC9)

NSD now displays the version of keyview associated with the current install of the Notes Client or Domino server in the NSD header. This is read directly from the Formats.ini located in the program directory of the Notes/Domino installation. For example:

Host Name : Localhost
User Name : SYSTEM
Date : Thu Jul 01 14:52:18 2010
Windows Dir : C:\Windows
Arguments : "C:\Program Files (x86)\IBM\Lotus\Notes\nsd.exe" -dumpandkill -termstatus 1 -dlgopts showwait -wctpid 2824 -wctexitcode 8096 -panicdirect -crashpid 3580 -crashtid 1112 -runtime 300 -ini "c:\Program Files (x86)\IBM\Lotus\Notes\notes.ini" -svcreq 128
NSD Version : 8.5.20.0137 (Build V852_05312010)
OS Version : Windows/7 6.1 [64-bit] (Build 7600), PlatID=2, (2 Processors) Running as 32-bit Windows application on 64-bit Windows
Build time : Tue Jun 1 01:56:39 2010
Latest file mod : Tue May 18 10:42:06 2010
Notes Core Version : Build V852_05312010NP (32-bit client)
Notes Standard Version : 8.5.2_20100602.0100
Keyview Version : 10.8.0.0

Adds support for detecting an notes2 abnormal termination on the Linux client (NBRR7V3N2S)

If the lnotes process (or nlnotes process, under Windows) recognizes that the notes2 process exited unexpectedly, then it will panic. With version 8.5.1, functionality was introduced under Windows that recognizes this situation, outputs a message to the NSD logging this condition, and then searches the workspace/logs directory for the presence of a Java^TM core file that may have been created if the notes2 process crashed.

This information would then be sent to the fault reporting database, if enabled by the administrator.

Now NSD will recognize this situation, if the -wctpid flag is present as a command line argument to NSD. This will be passed to NSD automatically by the fault recovery code. Example output from the NSD:

WARNING: NSD ran because the notes2 process exited unexpectedly (pid=4109).
The fatal thread reported will show only the panicking thread that detected that notes2 was missing.
Searching for javacore under /home/<username>/lotus/notes/data/workspace/logs...

If the javacore is found, this message displays:

"The javacore file /home/<username>/lotus/notes/data/workspace/javacore.txt was found. Please review this file for further information on why notes2 exited unexpectedly."

If it is not found, this message displays:

"Please inspect the workspace/logs directory under your Lotus Notes data directory for possible jvm dump, heap dump, or system core files."

Log the exit code of notes2 process and suspected reason in the event of a notes2 abnormal termination (NBRR83LMBV)

In a Standard Notes Client installation, if a notes2 abnormal termination is detected, NSD will record the exit code of the notes2 process via the -wctexitcode command line argument. This can be used to focus the message logged in the NSD to include the suspected reason that the notes2 process could have abnormally terminated.

For example, the NSD below was generated when the notes2 process was killed via the Task Manager, which caused the process to return with an exit code of 1. (Note that other scenarios could potentially cause the notes2 process to have an exit code of 1 also):

Host Name : Localhost
User Name : SYSTEM
Date : Wed Jul 07 11:32:56 2010
Windows Dir : C:\Windows
Arguments : "C:\Program Files (x86)\IBM\Lotus\Notes\nsd.exe" -dumpandkill -termstatus 1 -dlgopts showwait -wctpid 3500 -wctexitcode 1 -panicdirect -crashpid 3120 -crashtid 4924 -runtime 300 -ini "C:\Program Files (x86)\IBM\Lotus\Notes\notes.ini" -svcreq 128
NSD Version : 8.5.20.0137 (Build V852_05312010)
OS Version : Windows/7 6.1 [64-bit] (Build 7600), PlatID=2, (2 Processors)
Running as 32-bit Windows application on 64-bit Windows
Build time : Tue Jun 1 01:56:39 2010
Latest file mod : Tue May 18 10:42:06 2010
Notes Core Version : Build V852_05312010NP (32-bit client)
Notes Standard Version : 8.5.2_20100602.0100
Keyview Version : 10.8.0.0

WARNING (0): NSD ran because the notes2 process exited unexpectedly (pid=3500) with exit code 1. This indicates that the notes2 process was terminated via the Windows Task Manager.
Please inspect the workspace\logs directory under your Lotus Notes data directory for possible jvm dump, heap dump, or system core files.

The following NSD was generated when the notes2 process encountered an access violation. This resulted in a javacore being generated. The exit code is not predictable in the case of an access violation, so a general message is logged:

Host Name : Localhost
User Name : SYSTEM
Date : Thu Jul 01 14:52:18 2010
Windows Dir : C:\Windows
Arguments : "C:\Program Files (x86)\IBM\Lotus\Notes\nsd.exe" -dumpandkill -termstatus 1 -dlgopts showwait -wctpid 2824 -wctexitcode 8096 -panicdirect -crashpid 3580 -crashtid 1112 -runtime 300 -ini "c:\Program Files (x86)\IBM\Lotus\Notes\notes.ini" -svcreq 128
NSD Version : 8.5.20.0137 (Build V852_05312010)
OS Version : Windows/7 6.1 [64-bit] (Build 7600), PlatID=2, (2 Processors) Running as 32-bit Windows application on 64-bit Windows
Build time : Tue Jun 1 01:56:39 2010
Latest file mod : Tue May 18 10:42:06 2010
Notes Core Version : Build V852_05312010NP (32-bit client)
Notes Standard Version : 8.5.2_20100602.0100
Keyview Version : 10.8.0.0

WARNING (0): NSD ran because the notes2 process exited unexpectedly (pid=2824) with exit code 8096. The javacore file C:\Program Files (x86)\IBM\Lotus\Notes\Data\workspace\logs\javacore.20100701.145217.2824.0001.txt was found. Please review this file for further information on why the notes2 process exited unexpectedly.

This functionality is available only on Windows.

NSD -HANG -REPEAT -DELAY feature (work item 10890.04)

In order to make hang data collection more user friendly, the -hang switch was implemented in 8.5.2. This asks NSD to collect back-to-back native stacks and Java cores to help in the diagnosis of client hang situations. Once this data is generated, the client is optionally killed. If fault reporting is enabled, this data will be mailed to the fault report database the next time the client starts.

By default the stack dumps are generated twice, after a delay of 15 seconds. The number of times information is dumped and the length of time between dumps is configurable via the -repeat and -delay switches, respectively:

Usage example: nsd -hang -repeat=3 -delay=10000 -kill

• -hang = collect hang data
• -repeat = the number of times thread stacks and java cores will be collected (optional)
• -delay = the number of milliseconds to wait between each collection (optional)
• -kill = kills the Notes client processes (optional)

This also correlates to a new menu item that is available under Windows, Mac, and Linux (see figure 1).

Figure 1. New menu option

Adds support for data collection using jstack and sample on OS X (CMAS7ZGS3A)

Under OS X, the Standard Notes Client runs as a single process under eclipse.exe. Starting in 8.5.2, if NSD is run from the command line, it will run jstack so that Java thread stacks can be captured. To do this the sudo command must be executed, so the user is prompted for the administrator password.

When run from the command line, NSD will use jstack to collect Java thread stacks. Example output:

Attaching to process ID 238, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_22-147
Deadlock Detection:

No deadlocks found.

Thread t@64771: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=185 (Interpreted frame)
- org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=77, line=217 (Compiled frame)
- org.eclipse.core.internal.jobs.Worker.run() @bci=381, line=51 (Interpreted frame)

Thread t@68867: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=185 (Interpreted frame)
- org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=77, line=217 (Compiled frame)
- org.eclipse.core.internal.jobs.Worker.run() @bci=381, line=51 (Interpreted frame)

...

NSD will now use the sample utility to collect native thread stacks in the event of a crash and when NSD is executed manually. NSD previously used gdb to collect call stacks. Example output:

Attaching to /Applications/Notes.app/Contents/MacOS/rcp/eclipse/plugins/com.ibm.rcp.base_6.2.2.20100308-0530/macosx/x86/eclipse 238 (Wed Jul 7 13:08:31 CDT 2010)

...

High-profile fixes in Notes/Domino 8.5.2

Now let's go over the list of important fixes in 8.5.2.

NBRR7YBTQ2: Addresses a crash while collecting platform statistics on the Linux platform

On Linux a number of customers reported a crash with the following call stack:

Thread -1805739120 (LWP 17787)):
#0 0x004fd402 in __kernel_vsyscall ()
#1 0x0029da31 in select () from /lib/libc.so.6
#2 0x00670c50 in FRDoSleep ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#3 0x00671e55 in OSRunExternalScript ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#4 0x006734fc in OSFaultCleanupExt ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#5 0x006735e6 in OSFaultCleanup ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#6 0x0063fa67 in fatal_error ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#7 <signal handler called>
#8 0x006b74ca in LINUXPROCINFO_T::GetInfo ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#9 0x006b7ee8 in PROCINFO_T::GetStatus ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#10 0x006b73b9 in UnixProcessStats ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#11 0x006b18ab in PSGetDominoProcessInfo ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#12 0x08124ca1 in PSMain_orig ()
#13 0x08124cdb in PSMain ()
#14 0x081252d6 in PlatformStatsTask ()
#15 0x08070748 in Scheduler ()
#16 0x006686d6 in ThreadWrapper ()
from /opt/ibm/lotus/notes/latest/linux/libnotes.so
#17 0x0034d45b in start_thread () from /lib/libpthread.so.0
#18 0x002a4c4e in clone () from /lib/libc.so.6

The above stack indicates that Domino crashed while reading process information from the /proc file system. This issue was never reproducible in house; however, a code review of this area revealed a potential defensive fix, which was provided to several customers. The crash stopped occurring for these customers, so the issue was declared fixed.

JPAI849TL3: Skip _sminstal.exe issuing an nsd -kill

During the Smart Upgrade process, if the Notes Client does not exit cleanly or is unresponsive, the Smart Upgrade process ( _sminstal.exe) will run nsd -kill to bring down the client. In some instances, NSD could kill the Smart Upgrade process, so if NSD picks up _sminstl.exe as part of the current Notes instance, it will not kill the process. The implication is that nsd.exe will not kill this process, so if this process needs to be terminated, it must be killed manually.

JPMS7X3RDX: Disable the collection of Windows application/system/security logs during a nsd -info

When the Notes Client or Domino server starts, it executes an nsd -info to collect a snapshot of system information. Under Windows, NSD was crashing, dumping out the System Event logs as part of this process. This problem was never reproducible in house and only occurred in the field when nfileret was running an nsd -info.

System Event logs are noncritical system information, that are accessible via Windows Event Viewer if needed. Dumping this information also prolongs the amount of time that NSD takes to run. Crash stacks would be seen throughout the NSD log. For example:

ERROR (0): NSD thread 1408 got system exception: ACCESS_VIOLATION (3221225477)
### INTERNAL STACK DUMP BEGIN [ nsd: 03b4: 1408]
### @[ 1] 0x004af9c6 nsd.MEMHDR_T::IsAllocated+18 (4616D726F666E21,7720471c,11c670,236ae0)
### @[ 2] 0x004b119c nsd.memw_free+92 (4616D726F666E49,0,0,0)
### @[ 3] 0x004b12cb nsd.operator delete+31 (4616D726F666E49,11c6f0,1420,3633f98)
### @[ 4] 0x004aecc4 nsd.SYSLOG_T::GetDescription+2044 (12d958,46b6c78,0,11ce28)
### @[ 5] 0x004ad6b5 nsd.SYSLOG_T::DumpSysLog+1805 (12d958,870c08,530304100000004,30303000000011)
### @[ 6] 0x004417c1 nsd.NSDPARSER_T::SYSLOGSECSEC_T::GetData+149 (359e558,870c08,0,359e418)
### @[ 7] 0x0041d1a5 nsd.NSDPARSER_T::SECINFO_T::WriteData+137 (359e558,0,100000000,1)
### @[ 8] 0x00441840 nsd.NSDPARSER_T::SYSLOGSECSEC_T::DumpSysLog+48 (359e558,10,4,48)
### @[ 9] 0x00479b04 nsd.NSD_T::ShowSysInfo+3760 (870660,0,0,0)
### @[10] 0x00475f61 nsd.NSD_T::MainWrapped+5701 (870660,870c00,0,870e50)
### @[11] 0x00473df8 nsd.NSD_T::Main+6820 (870660,8,345a60,5b855c)
### @[12] 0x00470adc nsd.main+188 (8,345a60,0,0)
### @[13] 0x005c21d8 nsd.mainCRTStartup+568 (0,0,0,0)
### [14] 0x7721be3d kernel32.BaseThreadInitThunk+13 (0,0,0,0)
### [15] 0x77356a51 ntdll.RtlUserThreadStart+33 (0,0,0,0)
### INTERNAL STACK DUMP END

ERROR (0): NSD thread 1408 got system exception: ACCESS_VIOLATION (3221225477)
### INTERNAL STACK DUMP BEGIN [ nsd: 03b4: 1408]
### @[ 1] 0x004af9c6 nsd.MEMHDR_T::IsAllocated+18 (920656D69542D46,7720471c,11c670,237af0)
### @[ 2] 0x004b119c nsd.memw_free+92 (920656D69542D6E,0,0,0)
### @[ 3] 0x004b12cb nsd.operator delete+31 (920656D69542D6E,11c6f0,1420,46e9e98)
### @[ 4] 0x004aecc4 nsd.SYSLOG_T::GetDescription+2044 (12d958,46b6c78,0,11ce28)
### @[ 5] 0x004ad6b5 nsd.SYSLOG_T::DumpSysLog+1805 (12d958,870c08,530304100000004,30303000000011)
### @[ 6] 0x004417c1 nsd.NSDPARSER_T::SYSLOGSECSEC_T::GetData+149 (359e558,870c08,0,359e418)
### @[ 7] 0x0041d1a5 nsd.NSDPARSER_T::SECINFO_T::WriteData+137 (359e558,0,100000000,1)
### @[ 8] 0x00441840 nsd.NSDPARSER_T::SYSLOGSECSEC_T::DumpSysLog+48 (359e558,10,4,48)
### @[ 9] 0x00479b04 nsd.NSD_T::ShowSysInfo+3760 (870660,0,0,0)
### @[10] 0x00475f61 nsd.NSD_T::MainWrapped+5701 (870660,870c00,0,870e50)
### @[11] 0x00473df8 nsd.NSD_T::Main+6820 (870660,8,345a60,5b855c)
### @[12] 0x00470adc nsd.main+188 (8,345a60,0,0)
### @[13] 0x005c21d8 nsd.mainCRTStartup+568 (0,0,0,0)
### [14] 0x7721be3d kernel32.BaseThreadInitThunk+13 (0,0,0,0)
### [15] 0x77356a51 ntdll.RtlUserThreadStart+33 (0,0,0,0)
### INTERNAL STACK DUMP END

To address the problem, nsd -info no longer dumps system event logs. If the collection of this data is desired, a user can explicitly enable it by creating the following entry in the nsd.ini.

syslog=1

HKAI7RB9CF: Addressed an NSD crash that occurred while dumping the OS Process Table

In order to print the command line arguments for each process in the process table, the command line data that is passed as an argument to the process on startup must be read from the process environment block. In some instances this was a NULL pointer, which NSD would then try to print out and crash with the following stack:

ERROR (0): NSD thread 14a4 got system exception: ACCESS_VIOLATION (3221225477)
### INTERNAL STACK DUMP BEGIN [ nsd: 14bc: 14a4]
### @[ 1] 0x004c9db0 nsd.strlen+48 (0,f7,12e3f8,12e3dc)
### @[ 2] 0x0049b87a nsd.STR_T::EllipsisCopy+26 (12e17c,f7,0,0)
### @[ 3] 0x0049b7ca nsd.GetProcessCommandLine+730 (acc,12e3f8,f7,12e45c)
### @[ 4] 0x004086b0 nsd.THRNAME_T::Format+272 (0,acc,0,0)
### @[ 5] 0x00408ab0 nsd.WINTHRNAME::WINTHRNAME+80 (0,acc,0,0)
### @[ 6] 0x0041842d nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProc+221 (5d1104,508e2f,e5d5ac,6)
### @[ 7] 0x004185ad nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTree+141 (5d1104,e5d5ac,6,1)
### @[ 8] 0x0041862a nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTree+266 (5d1104,e5c314,5,1)
### @[ 9] 0x0041862a nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTree+266 (5d1104,e2f144,4,1)
### @[10] 0x0041862a nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTree+266 (5d1104,e26424,3,1)
### @[11] 0x0041862a nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTree+266 (5d1104,e24784,2,1)
### @[12] 0x0041862a nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTree+266 (5d1104,e2428c,1,1)
### @[13] 0x0041862a nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTree+266 (5d1104,e23dbc,0,1)
### @[14] 0x00418303 nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::GetData+163 (5d1104,0,3ff12c,5d1104)
### @[15] 0x00433879 nsd.NSDPARSER_T::SECINFO_T::WriteData+57 (0,0,3ff12c,12e77c)
### @[16] 0x0043570f nsd.NSDPARSER_T::WINOSPROCTABLESEC_T::DumpProcTable+63 (1,0,5d0d38,12ec58)
### @[17] 0x0043569c nsd.NSD_T::ShowProcs+92 (1,0,0,ece)
### @[18] 0x004331bc nsd.NSD_T::MainWrapped+2892 (7c801dc6,0,154b90,3f0000)
### @[19] 0x00431e9a nsd.NSD_T::Main+5066 (4,3f2c28,0,ece)
### @[20] 0x0042f753 nsd.main+147 (4,3f2c28,3f2cc0,94)
### @[21] 0x004c980f nsd.mainCRTStartup+364 (0,0,7ffde000,0)
### [22] 0x7c82f23b kernel32.ProcessIdToSessionId+521 (4c96a3,0,78746341,20)
### INTERNAL STACK DUMP END

This problem was never reproducible in house. It is suspected that the problem occurred in a specific scenario when a process was spawned via a service.

NBRR7ZDQT4: Addressed an issue where thread ids are reported as negative in semaphore debugging

Thread IDs present in semaphore debug showed as negative integers. This issue was reported on 64-bit UNIX platforms, but could occur on any platform.

sq="0000B504" THREAD [15289:00273-3989617568] WAITING FOR WRITE LOCK ON FRWSEM 0x4245 open database queue semaphore (@F5596C40) (R=1,W=0,WRITER=00000:00000,1STREADER=29357:-859944032) FOR 5000 ms

ti="00717F14-C1257689" sq="000002FE" THREAD [29353:00048-3718753184] WAITING FOR READ LOCK ON FRWSEM 0x030B Collection semaphore (@F2D60F74) (R=0,W=1,WRITER=29353:-574628960,1STREADER=00000:00000) FOR 5000 m

ti="00271064-C125768A" sq="000013B1" THREAD [15289:00101-3984825248] WAITING FOR SEM 0x0944 Task table entry semaphore (@F0967CCC) (OWNER=15289:-307213408) FOR 5000 ms

In this instance the OWNER, READER, or FIRST READER fields of the semaphore debug showed as negative. Upon reviewing the code it was clear that other areas of semaphore debug also exhibited the potential for this problem and were corrected under this fix.

KUMA7VJJBB: Introduced an INI DEBUG_DISABLE_NOTES2_EXCEPTION_FILTER

The ini parameter DEBUG_DISABLE_NOTES2_EXCEPTION_FILTER was introduced as a defensive fix. This parameter prevents the notes2 process from registering with the Notes fault recovery code, which is responsible for running NSD in the event of a crash. Because the notes2 process is a JVM process, we would instead want the typical JVM crash data to be generated Java dumps, heap dumps, and system cores), not an NSD.

For example, if an NSD was found in which the fatal process listed was notes2.exe, then this process registered with the Notes fault recovery code at some point in its life cycle. This is not desired behavior. The main reason this could happen is via a plug-in that uses the Notes API, but this was not reproducible in typical usage of the Notes API. Unknown circumstances are required to end up in this condition.

In 8.5.1FP1 if this parameter is set to 1, it explicitly prevents the notes2 process from registering with the Notes fault recovery code. In 8.5.2, this is on by default, and DEBUG_DISABLE_NOTES2_EXCEPTION_FILTER =0 must be set to revert to the default behavior.

It should also be noted that if an NSD is produced, instead of a javacore, nlnotes will detect that the notes2 processes exited unexpectedly and will panic. If this occurs, another NSD will be generated, reporting a "WCT Abnormal Termination" along with the fatal crash stack associated with this condition:

############################################################
### thread 7/26: [ NLNOTES: 0a5c: 091c] FATAL THREAD (Panic)
### FP=0x0525dbb4, PC=0x7c90e514, SP=0x0525db50
### stkbase=0x05260000, total stksize=262144, used stksize=9392
### EAX=0x00000000, EBX=0x0525e784, ECX=0x7ffd5000, EDX=0x00270608
### ESI=0x000006f0, EDI=0x00000000, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297
############################################################
[ 1] 0x7c90e514 ntdll.KiFastSystemCallRet+0 (6f0,493e0,0,525e364)
[ 2] 0x7c802532 kernel32.WaitForSingleObject+18 (6f0,493e0,525eaf8,525e7a9)
@[ 3] 0x601b4ec5 nNOTES.FRSendCommandToService+789 (525e384,525e784,525e7a9,0)
@[ 4] 0x601b5aff nNOTES.OSRunExternalScript@8+1055 (12c,1)
@[ 5] 0x601b607f nNOTES.FRTerminateWindowsResources+975 (1,1010,1,0)
@[ 6] 0x601b64a8 nNOTES.OSFaultCleanupExt@24+984 (c76a68,1010,0,0,0,525ee20)
@[ 7] 0x601b652a nNOTES.OSFaultCleanup@12+26 (0,1010,0)
@[ 8] 0x601c1a24 nNOTES.OSNTUnhandledExceptionFilter@4+276 (525fe58)
@[ 9] 0x6018477d nNOTES.Panic@4+589 (525fe70)
@[10] 0x60184834 nNOTES.OSWCTPanicShutdown+84 (a74,1232ee8,12357ee,3)
@[11] 0x63680da3 nnotesws.WCTShutdownThread+211 (a74,138390,7c910041,0)
@[12] 0x6010dabf nNOTES.ThreadWrapper@4+175 (0)
[13] 0x7c80b699 kernel32.GetModuleFileNameA+442 (0,0,0,0)

There has been some confusion behind this parameter in thinking that it fixed the above fatal stack. It does not. This fatal stack will occur whenever the notes2 process abnormally terminates, in which case the underlying cause of the abnormal termination must be discovered and addressed.

DLUI7RYB2A: Addressed an issue where NSD would search for AWK using AWK, causing nsd.sh to loop infinitely

This fix checks for AWK, and exits with the following message if it is not present:

"ERROR: NSD couldn't find a necessary dependency (awk). For NSD to function, awk (or a soft link to it), should be located in /usr/bin, /bin, /sbin, /usr/sbin, or /usr/ucb."

Previously, NSD would attempt to check for awk, with a search that used awk, causing an infinite loop if it wasn't present.

LHAR7W4AHV: Addded the -nodirlist parameter to nsd.sh for Windows/UNIX parity of command line options

As part of its output, NSD recursively lists the contents of the data, program, and transaction logging directories. This can sometimes take a long time and occupy a significant chunk of space in an NSD.

To disable this option under Windows the -nodirlist option can be passed to NSD on the command line, or nodirlist=1 can be specified in the nsd.ini. Under UNIX the -nofs switch performs the same operation; however, the nodirlist switch was added under UNIX to maintain parity between common Windows and UNIX nsd functions.

About the author

Nikolaus Brauer is a Level 3 Notes/ Domino Serviceability Lead currently based at IBM's Austin, TX, Support Center.